1. Field of the Invention
The present invention relates to an image processing apparatus, image sensing apparatus, control method, and recording medium, and particularly to a technique of generating, from raw data after shooting, an image focused on at an arbitrary focal plane.
2. Description of the Related Art
There has recently been proposed a technique of recording the intensity distribution and traveling direction of light as raw data upon shooting, and generating, from the recorded raw data, an image focused on at an arbitrary focal plane in an image sensing apparatus such as a digital camera.
“‘Light Field Photography with a Hand-Held Plenoptic Camera’, Stanford University Computer Science Tech Report CSTR 2005-02 by Ren Ng, et al.” discloses a method (Light Field Photography) of forming images from light beams having passed through different pupil regions of an imaging lens on the respective pixels of an image sensor via a microlens array, thereby splitting and recording light beams entering from various directions. In obtained raw data (light field data), adjacent pixels record light beams entering from different directions. By extracting light beams in the same direction from pixels corresponding to respective microlenses, an image shot from this direction can be generated. Further, an arbitrary focal distance is set, and outputs from pixels which record light beams having passed through one point (corresponding to a pixel) on the focal plane (refocus plane), in which the optical distance from the shooting lens is corresponding to this focal distance, are added. This enables pseudo-generating the pixels of an image focused on at a specific focal distance after shooting. Hereinafter, if the user of the image sensing apparatus intends to focus to other subject after obtaining the light field data, “deciding a focal distance of an imaging optical system in order to focus the other subject, and generating, from the light field data, an image for a refocus plane defined by an optical distance (a distance between a point, at which light beams from the other subject are focused on, and an imaging lens) corresponding to the focal distance” is just referred as “generating an image focused on at the focal distance (corresponding to the other subject)”.
The method in “Light Field Photography with a Hand-Held Plenoptic Camera” is also applicable to recording of a moving image. More specifically, a moving image focused on at an arbitrary focal distance can be generated from raw moving image data after shooting.
In a moving image, a subject moves during shooting. Hence, in moving image shooting by a general image sensing apparatus, the focal distance of the optical system is changed during shooting in accordance with the movement of a subject to be focused on.
However, when generating a moving image focused on at an arbitrary focal distance from raw moving image data recorded using a method as described in “Light Field Photography with a Hand-Held Plenoptic Camera”, there has not been proposed a method of changing a focal distance in accordance with the movement of a subject to be focused on.